Coordination and Fusion in Multimodal Interaction

Multimedia Input Analysis Whereas traditional interfaces support sequential and unambiguous input from devices such as keyboard and conventional pointing devices (e.g., mouse, trackpad), intelligent multimodal interfaces (see www.mitre.org/resources/ centers/it/maybury/tutorial.html) relax these constraints and typically incorporate a broader range of input devices (e.g., spoken language, eye and head tracking, three dimensional gesture). For example, they support asynchronous, ambiguous, and inexact input by applying more sophisticated analysis of input. These systems allow the resolution of multimedia references, for example enabling the user to say "Put that there" while gesturing to a map, by correlating eye and hand gestures with the deictic expressions “that” and “there” (Burger and Marshall 1993). Intelligent interfaces can also exploit models of the media, user, discourse, and task and automatically detect and correct errors.